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ABSTRACT 

Judgment ANalysis (JAN) was presented as a technique for capturing policy(ies) 
present in a group of decision-makers. Several studies were described in which the 
jSi^ JAN methodology was successfully utilized in capturing strategies of raters with 
I respect to the responses to a set of complex stimuli. Examples were given which 
I suggest that the JAN methodology was effective not only in the area of making 

definitions explicit but also in the evaluation of programs and research proposals. 

1. Judgment Analysis 

The objective of this paper is to describe a policy-capturing procedure. Judgment 
ANalysis (JAN), which can make explicit how judges (decision-makers) make decisions 
under a variety of conditions and to illustrate how the technique can be extended not 
only to the area of making definitions explicit but also to the evaluation of programs 
and research proposals. 

A technique for both capturing and clustering raters' policies. Judgment 
Analysis (JAN), was suggested by Bottenberg and Christal (2) and Christal (4,5) 
and has been described in some detail by Houston and Stock (19). JAN utilizes a 
multiple regression model and a hierarchical grouping technique which clusters raters 
on the basis of the homogeneity of their raw score regression vectors (their prediction 
equations). In the use of JAN members of a group of decision-makers are required to 
rank or rate stimuli on a criterion. Results of JAN permit the identification of the 
rating policy (or policies) txisting in the group or committee of judges and the number 
of clusters (and its members) of decision policies, if more than one policy, that 
exist for the total group. The end product is the systematic use of the resulting 
prediction equaition or equations to simulate either group judgment or individual 
policy as well as to enable the investigator to make a complete analysis of interrater 
agreement . 

The JAN procedure basically consists of two separate stages. During the first 
stage of the JAN process, a least-squares solution of a multiple regression equation 
for each rater separately is computed which predicts to some degree the criterion 
responses or decisions made by the judge. The results of stage one provide the 
investigator with the specification of each judge's policy (decision-making equation) 
and a measure of its reliability or consistency (r2 value). Given a specific set of 
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independent variables the investigator is able, after stage one, to predict with a 
specific degree of confidence how a particular judge will make his decisions for a 
given set of profiles or stimuli. In other words, the question of whether the^ 
decision-makers are consistent in their use of independent variable? as opposed to 
unreliable (a policy with decisions expressed in some stochastic fashion) can be answered 
from the results of stage one. In addition, a statement of the policy for each judge 
can be determined. 

After the results from stage one are obtained, a hierarchical grouping procedure 
is initiated. This procedure allows the investigator to group individuals according 
to some objective function. In this case, groupings are sought which yield the 
smallest decreasein the ability of the system to predict the criterion responses from 
knowledge of the individuals who served as judges. The first step in the grouping 
procedure reduces the N equations (a separate equation for each of N judges) to N-1 
equations. In the first reduction of equations, the two judges who are most alike 
in their policies are grouped together and all other judges remain ungrouped. The 
process continues in a systematic way reducing the number of judges by one at each 
step until ail judges have been grouped into a single cluster. At each utage of the 
grouping an examination of the loss of predictive efficiency makes it possible to 
identify the different judgmental policies which exist. Results of JAN after stage 
two permit the identification of the rating policy (or policies) existing in the group 
or committee of judges and the number of clusters (and its members) of decision policies, 
if more than one policy, that exist for the total group. Policy equation (or equations) 
can then be used to simulate either group judgment or individual policy. The 
investigator can also make a complete analysis of interrater agreement. {See Ward 
and Jennings (29) for the full general linear model and successive restricted models 
necessary for the hierarchical clustering of judges at each stage of the JAN clustering.} 

Often, after stage two, it is found that more than one Policy exists in a group 
of decision-makers. If the primary purpose of the study ib to develop a single joint 
or collective policy as opposed to analyze the several policies which may be present, 
the following course of action is recommended. Present to the decision-makers 
results from stage two, i.e., make explicit the different policies which were captured 
and identified by means of the JAN technique. Have the group of assembled decision- 
makers arbitrate the rating or rank responses on profiles in which disagreement has 
resulted in policy differences. The technique of arbitration of response decisions 
rather than of the weighings applied to the independent variables is one of the powerful 
aspects of this technique. Once an arbitration of the response decisions has been 
accomplished, a single joint policy may be obtained by regressing the arbitrated 
responses on the set of independent variables. {See Christal (4, 5) who discusses 
this aspect of the procedure in greater detail.} 

2. General Applications of Judgment Analysis 

JAN has already been demonstrated to be an extremely useful methodology for 
describing and capturing the strategies of raters with respect to the responses to a 
set of complex stimuli. JAN has been used in several studies conducted by the U.S. 
Air Force for job evaluations and to simulate officer promotion boards with a high 
degree of efficiency (4, 5). Equations have also been designed to simulate career 
counselors in making initial assignments of airmen graduating from basic training (5). 
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The JAN technique has been applied in a prediction study of success in graduate 
education. In this study two variations of JAN were investigated by Houston (13) 
and Roscoe and Houston (23) --Normative JAN and Ipsative JAN. The purpose of the 
Normative JAN study was to determine the extent to which a policy regarding graduate 
admission standards existed among selected graduate faculty members at Colorado 
State College (now University of Northern Colorado). Basically, three sets of 
independent profile variables were used: (1) biographical data, (2) test data, and 
(3) major subject field data. Results from the Normative JAN study indicated 
essentially one policy was present in the group of judges. 

The Ipsative JAN study used for its dependent variable the rankings submitted 
by the judges who were requested to rank, without access to the three sets of in- 
dependent profile variables used in the Normative JAN study, the doctoral graduates on 
the basis of a personal knowledge. It was the intent in this phase that the ratings 
or rankings be loaded with personality factors not readily available in the Normative 
JAN study. Results of this phase were statistically significant, though weak from a 
predictive standpoint. The practical significance of the Ipsative JAN study was in 
the suggestion of new directions for subsequent research. 

Williams, Gab, and Linden (3) replicated Houston's Normative study at the 
University of North Dakota and sought to determine the policy of a university doctoral 
admissions board. Twelve members of the graduate faculty evaluated each graduate 
student's profile and replaced it into one of seven criterion categories (Q-Sort). 
Each rater's policy was assessed or captured and the raters were grouped into appropriate 
clusters by the JAN process. They found that at least two separate judgmental systems 
were present . 

A further illustration of the versatility of the technique is provided in a 
study by Stock (24) who sought to determine if systematic differences existed in 
the placement policies for special education students among special education personnel 
(teachers, administrators, and members of the special education screening committee) 
responsible for placing the students in the public schools of Cheyenne, Wyoming. 
Coltvet (6) used JAN techniques in the identification and analysis of the consultant 
ratings of elementary student teachers at the University of Northern Colorado. The 
policies involv^ed in the selection of faculty members at two small private colleges 
were investigated by Heim (11). Using JAN procedures, Chang (3) designed a study to 
determine whether individuals serving in different official capacities in the State 
of Colorado had differing attitudes toward selection criteria for awarding college 
financial grants. Keelan, et al . (21) captured the leadership policies of selected 
firemen in the State of Colorado with the use of JAN. 

The policy-capturing methodology has been extended to cover the situation in 
which raters are required to make multidimensional rating responses, i.e., they must 
rank or rate each profile on more than one output dimension. Johnson and King (2) 
developed as part of a team dissertation a multivariate clustering procedure which 
captures multidimensional policies expressed by k- judges. The technique was identified 
as Canonical Judgment Analysis (C-JAN) and it represents an extension of Judgment 
Analysis procedures to two or more dimensions. 
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3. Making Dafinitions Explicit with JAN 

For nearly as long as organized society has existed there has prevailed concern 
and confusion as to what is pornographic or obscene. This confusion has greatly 
increased va the United States in the last decade as a result of the vague and weakly 
defined deciS.lons handed down by the Supreme Court, Because of these poorly framed 
decisions it is not surprising that most people, professional and lay, are uncertain 
as to what constitutes pornography. Dr, Joe Ward suggested that the JAN technique 
might be used in this controversial area to assist those confronted with making 
decisions about what is pornographic. J. Houston and S. Houston (2) used JAN as a 
methodology by testing this technique with three groups concerned with this issue. 
These groups included doctoral students majoring in Psychology, Counseling and Guidance 
at the University of Northern Colorado, lawyers and police officers from the city of 
Greeley, Colorado. Each judge (who served on a voluntary basis) was presented with 
a sample of 100 profiles, generated for this study by the investigators, descriptive of 
pictorial representations. Each judge was requested to rate each of the profiles on 
a five-point scale. 

The instrument used to gather the data contained an 11-item profile for each of 
100 pictorial representations. The 11 variables and a sample profile appears below: 

Sample Profile of Pictorial Representation 



Descriptors Yes No 

1. Appeals to prurient interest X 

2. Goes beyond contemporary community standards X 

3. Has redeeming social importance X 

4. Involves uncovered adult human genitalia X 

5. Displays human sexual intercourse X 

6. Displays masturbation X 

7. Displays anal intercourse X 

8. Displays flagellation or torture X 

9. Displays homosexuality X 

10. Involves interracial sexual relationships X 

11. Displays group sex activity X 



Judge's Rating (Circle only one) 12 3 4 5 

Code 

1. Should be banned from society 

2. Allowed only for professional (medical and/or scientific) study 

3. Available only to interested adults 

4. Available to all adults 

5. Unrestricted use in society 

The reason for using 11 items on each profile and having 100 profiles available 
for each of the judges stems from the research conducted by Dudycha (9). More specifically, 
Dudycha found that it was difficult to cluster judges or raters when fewer than 100 
observations existed. Further, Dudycha established that the clustering technique is 
far more effective when at least 10 predictor or profile items are used. 
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Each of the 100 profiles was generated by employing a table of random numbers. 
The table of ranc'om pumbers was used to insure that the variables would not be system- 
atically correlated; thus, the problem of multicollinearity would be avoided. This 
technique guaranteed also maximum variability in the distribution of profile descriptors, 
which is essential for the valid use of JAN. 



The JAN technique proved to be surprisingly effective in capturing and clustering 
the policies (specific and complex) of the judges from the three groups identified. 
As expected, many policies were present. Although the process of identifying porno- 
graphic material is not a simple one, the JAN technique can be successfully utilized 
in capturing and explaining different policies regarding the pornographic content of 
material. Such a process as Judgment Analysis will at least open up a new frontier 
when psychologist, legal authorities, and government officials attempt to formulate 
policies regarding ''what is pornographic and what is not." The investigators believe, 
however, that further investigations utilizing the JAN technique are needed. Such 
studies should involve a greater cross-section of professional and lay peoples, a 
larger geographic representation and a greater, more finite variety of pornographic 
material . 



4. Evaluation Applications and JAN 



The JAN methodology shows great promise in the area of education evaluation. The 
problem of evaluating curriculum packages was investigated by Torgunrud (25) in a 
doctoral dissertation completed at the University of California at Los Angeles. 
Torgunrud identified from the educational literature the following independent variables 
as important dimensions of any curriculum package or set of materials which are under 
consideration for possible adoption. These include: (1) valid and significant content 
(2) significant elements of organization, (3) sequence providing a cumulative effect, 
(4) integration providing horizontal relationships, (5) value position clearly stated, 
(6) specificity providing direction, (7) flexibility providing alternatives, (8) 
accommodation of student differences, (9) accommodation of teacher competencies, (10) 
accommodation for student participation, and (11) providing for measurement of 
achievement. After defining the variables, Torgunrud generated a sample of ICQ 
pr.. files, each described by Naylor and Wherry (22) for simulating stimuli with 
specified factor structure. An example of a typical profile appears below: 

Sample Curriculum Package 
Descriptor's Weak Average Strong 

123456789 



1. 

2. 

3. 

4. 

5. 

6. 

7. 

8. 

9. 
10. 
11. 
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Valid and significant content 
Significant elements of organis^ation 
Sequence providing a cumulative effect 
Integrating providing horizontal relationships 
Value position clearly stated 
Specificity providing direction 
Flexibility providing alternatives 
Accommodation of student differences 
Accommodation of teacher competencies 
Accommodation for student participation 
Providing for measure of achievement 

Judge's Overall Rating of Curriculum 
Package (Circle only one) 



. . . 3 

Weak Average Strong 

1 2 34567 89 
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Each of the participating judges (school superintendents) was requested to 
rare or rank each of the san^ple curriculum packages. The JAN procedures enabled the 
investigator to determine the relationships of the set of independent variables to 
the judge's specific policy relative to curriculum packages. In addition, a 
determination was made on how many different evaluation policies were actually present, 
who were the members expressing each policy, and what were the areas of agreement and 
disagreement in the different policies. 

In another evaluation study at the University of California at Los Angeles, 
Duff (10) utilized JAN techniques to capture both the teacher-hiring policies (Ex Ante ) 
of selected administrators and the administrators' evaluation policies (Ex Post ) 
of teachers' on-the-job performance after their first year of paid teaching experience. 
Both types of policies (hiring and job performance) were analyzed in terms of predictive 
validity by the investigator. 

The concept of a group or collective judge was introduced in a study by Houston 
and Gilpin (17) and in a follow-up study by Houston et al . (16) in which the investigators 
sought to capture the evaluation policies of selected groups of students who were 
rating faculty members at the University of Northern Colorado. Rather than use each 
student as a separate judge, students were classified by their grade level and by the 
school or college in which they were enrolled as a major. A copy of the instrument 
presented to the student is as follows: 



Teacher Description Instrument 



1) Poor 



2) Fair 3) Average 



accordance with this 


rating scale. 


4) 


Good 


5) 


Excellent 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 


1 2 


3 


4 


5 



1. Teacher's interest and enthusiasm for course 

2. Ability to answer questions adequately 

3. Ability to communicate the subject matter effectively 

4. Ability to interest and motivate students 

5. Fairness in testing and grading 

6. Personal interest and adaptation to student's needs 

7. Course Objectives are clearly stated 

8. Course objectives are met 

9. Everything considered, including strengths and 
weaknesses, I would rate the instructor 

The first eight items of the TEACHER DESCRIPTION INSTRUMENT were considered 
independent variables and item nine was treated as the dependent variable in the 
regression analyses which were completed. This study departed from other JAN studies 
not only in the use of a group judge but also in the use of the students to generate 
the profile scores (normally each judge receives a fixed set of profiles to rate or rank). 
In the first JAN study (JAN I) the students were grouped by five grade levels thus 
permitting the use of five judges in the JAN analysis. The judges for t\ e JAN I 
study included: Judge 1--Freshmnn students; Judge 2--Sophomore students; Judge 3-- 
Junior students; Judge 4--Senior students; and Judge 5--Graduate students. In 
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the second JAN study (JAN II) the students were identified in terms of the seven 
schools or colleges at the University of Northern Colorado. The judges for the JAN II 
study included: Judge 1--Students from School of the Arts; Judge 2--Students from 
College of Arts and Sciences; Judge 3 — Students from Schocl of Business; Judge 4-- 
Students from College of Education; Judge 5--Students from School of Health, Physical 
Education, and Recreation; Judge 6--Studenf:s from School of Music; and Judge 7-- 
Students from School of Nursing. In both studies it was found that only one policy 
was present. The overall rating of the faculty member was primarily a function of 
variables 3,4, and 6. 

Teacher effectiveness policies of faculty members were captured in a study by 
Houston and Broderius (15). Profiles of sixty hypothetical faculty members were 
generated in which each faculty member was described in terms of teaching skills, 
student advising skills, institutional service, and professional activities. Three 
policies were found to exist in the faculty members who participated in the study. 

Another potential application of JAN techniques is in the area of program evaluation, 
especially in the case involving the use of evaluation experts rendering professional 
judgments. Two research studies are described by Houston and Bentzen (14) and Houston 
et.al. (18) in which JAN was used as part of the summative evaluation of the projects. 
Both programs relied on the expertise of educational evaluation experts who were 
required to make ratings on several dimensions or variables describing the school or 
classroom p'-ogram. These variables were later used as independent predictors of the 
dependent variable (which was an overall rating of the school or classroom program) 
in the policy statement of each judge which was captured with the aid of JAN- With 
the assistance of JAN procedures the investigators were able not only to capture the 
evaluation policies present in the team of evaluation experts but also able to determine 
which of the independent variables were making significant contributions to the overall 
success of the projects. Instead of providing much input and output information 
separately, JAN was able to tie together and relate those input aspects of the project 
responsible for the ultimate impact of the program. Indeed, JAN can and does provide 
another dimension to program evaluation and is recommended for consideration in programs 
which rely heavily on professional judgments of experts in the final evaluation. 

5. Evaluation of Research Proposals and Judgment Analysis 

A very basic question in the evaluation of research proposals seeking funds is 
to determine which proposals should get supported. In answering this question a typical 
starting point is often to ask the team of evaluators to rate each proposal on a set of 
characteristics. An example of such a set of characteristics for proposals at the 
National Cancer Institute on which a team of evaluators would rate was presented 
by Dr. John C. Bailar, III, Deputy Direc*:or for Cancer Control, National Cancer Instittfte (1). 
The dimensions or characteristics identified include: (1) need, (2) relevance, (3) 
feasibility, (4) number benefitted, (5) how much they would benefit, (6) acceptability, 
(7) potential, (8) cost benefit, (9) ease of evaluation, and (10) originality. A 
question often left unanswered (or answered inadequately) when d-^ta are available 
which are descriptive of a specific proposal, is how are each of the ten dimensions 
to be weighted in making a decision to fund or not to fund the project. 
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A strategy for utilizing JAN in this situation would be to start with the set 
of 10 characteristics or dimensions and generate a simulated sample of approximately 
200 representative proposals possessing a required factor structure. (See the dissertation 
of Torgunrud (25) and the article of Naylor and Wherry (22) for a description of the 
procedure requirad to generate simulated stimuli. However, other approaches are 
available depending on the avilability of stimuli and the requirements of the factor 
structure.) Have each of the officials involved in the decision-making process 
rate each of the 200 proposals. An extremely difficult assignment for the official 
(who serves as judge or rater) would be the assignment that he rank order the complete 
set of 200. A more reasonable and practical request would be for each judge to place 
each of the 200 proposals into different ordered categories (usually a minimum of 
five). While original studies involving JAN requested each judge to rank order the 
complete set of profiles or proposals, experience with the technique has suggested that 
JAN is still quite effective when the assignment involves the placing of profiles into 
ordered categories, even as few as five categories. Judges find it too difficult to 
rank order a large sample of stimuli and they typically have little difficulty in 
making an assignment involving a few ordered categories. 

An example of a sample proposal or profile described on the 10 characteristics 
appears below: 

Sample Research Proposal 



Descriptors 

1. Need 

2. Relevance 

3. Feasibility 

4. Number benefitted 

5. How much they would benefit 

6. Acceptability 

7. Potential 

8. Cost benefit 

9. Ease of evaluation 
10. Originality 

Judge's Overall Rating of 
Proposal (Circle only one) 



Weak Average Strong 

123 4567 89 



. . 3 



Weak 
12 3 



Average 
^567 



Strong 
8 9 



After each of the officials serving as judges has completed the rating assignment 
for the 200 simulated stimuli, begin the JAN analysis. For a description of the 
computer program for JAN, see Vlahos and Houston (26). From the JAN print-out and 
using the recommendations by Ward and Hook (28) for determining the number of policy 
groupings, identify the number of captured policies and the judges associated with 
each of the policies. Using the knowledge of how many policies are present and who 
are the judges in the different policy groupings, the investigator should then attempt 
to explain each of the captured policies. A variety of approaches have been as< J to 
explain the different policies. These approaches, depending on the research question 
and the factor structure relating the set of independent variables, include path 
analysis (32), principal components analysis (7), stepwise regression analysis (8), 
setwise regression analysis (31), subjective hierarchical analysis (17), all possible 
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regressions (8), the backward elimination procedure or unique contribution 

analysis (8,27), the forward selection (8) and the stagewise regression procedure (8). 

Return to the judges gathered together and explain the different policies with 
particular emphasis on areas of agreement and disagreement. At this point identify 
specifically the profiles whose varied rankings resulted in the separate policies 
which were captured. Ask the group to arbitrate the rankings where differences have 
occurred, and not the weightings to be applied to the independent variables. After 
the group agrees on the rankings for the simulated proposals complete a new regression 
analysis using the new rankings as the dependent variable and thereby obtain the 
p4:oper weights associated with the joint policy. 

5. Summary 

Judgment ANalysis (JAN) was described as a vehicle for capturing policy (ies) 
present in a group of decision-makers. Several studies were presented which strongly 
support consideration of its use in the area of making definitions explicit as well 
as a technique for evaluating program and research proposals. Examples of how the 
JAN methodology might be applied to proposal and program evaluation were illustrated. 
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